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DETAILED ACTION 
Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill In the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1-3, 5-6, 8-10, and 39 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over SzellskI et al. U.S. Patent No. 6157747 In view of Luken U.S. 
Patent No. 5923334 and Seago U.S. Patent No. 5990900. 

Referring to claim 1, Szeliski et al. teaches a computerized method for creating 
a three dimensional model from image panoramas, the method comprising: 

receiving at a computer a plurality of image panoramas representing a visual 
scene and having an object (Figs. 1 , 3 and 36A-38; column 27, Lines 34-48, i.e. 
constructing a complete panoramic mosaic, converting the set of input images and 
associated transforms into one or more images which can be quickly rendered or 
viewed by choosing either a cylindrical or spherical map and converting a rotational 
panorama to a spherical panorama indicates that a plurality of panoramas are being 
received. Figure 6, element 610 and column 1 1 , lines 16-22 discloses wherein a 
deformation D provides a final correction such that the placement of a fixed object 
relative to the second image's re-warped coordinate system (x", y") is about the same 
as the object's placement relative to the first image's coordinate system (x, y) thus 
indicating that the plurality of image panoramas have an object as claimed). 
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the object occupying a field of view of more tlian 1 80 degrees in tine panoramas 
and having an object (Figs. 1 , 3, 6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 
31 , lines 51-64, i.e. one of ordinary skill in the art at the time of invention would 
recognize that the wall objects in figures 33A and 33B and the hotel lobby wall objects 
are both shown occupying a field of view of more than 1 80 degrees since Applicant 
discloses that panoramic images of wall structures indicate objects occupying a field of 
view of more than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph); 

using the computer, determining a directional vector indicating an orientation of 
the visual scene with respect to a reference coordinate system (Fig. 15; column 22, 
lines 4-31 , i.e. figure 1 5 shows the adjustment of the bundle of rays xjk so that they 
converge to xy. Letting the ray direction in the final composited image mosaic be a unit 
vector p, and its corresponding ray direction in the kth frame be py/f-R/f'V/f'^xy/f, indicates 
a directional vector for each image panorama as claimed); 

transforming the image panoramas such that the directional vectors are 
substantially aligned relative to the reference coordinate system (Fig. 15; column 22, 
lines 2-31 , i.e. figure 1 5 shows the adjustment of the bundle of rays xjk so that they 
converge to xy. Letting the ray direction in the final composited image mosaic be a unit 
vector p, and its corresponding ray direction in the kth frame be py/f~R/f'V/f"'xy/f, indicates 
transforming the image panoramas as claimed); 

aligning the transformed image panoramas to each other (Figs. 17-18 and 20; 
column 22, lines 46-67; column 23, lines 10-12; column 24, lines 30-61, i.e. global 
alignment of the image panoramas is being performed using patch-based alignment). 
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Szeliski et al. does not specifically teach determining a directional vector for each 
image panorama; and 

using the computer, creating a three dimensional model of the visual scene from 
the transformed and aligned image panoramas using the reference coordinate system, 
wherein creating a three dimensional model includes identifying at least one boundary 
of the object and using the identified boundary to associate geometry information with 
the object, the geometry information comprising 3-D coordinates describing the position 
and orientation of the object boundary in the reference coordinate system. 

Luken teaches eight direction vectors D0-D7 associated with six rectangular 
images mapped to the inside of an octahedron wherein it is determined which of the six 
rectangular images is intersected by one of the eight direction vectors (Figs. 7-10, 14 
and 17; Column 7, lines 28-36). 

Seago et al. teaches creating a three dimensional model of the visual scene from 
the transformed and aligned image panoramas using the reference coordinate system 
(Fig. 2; column 4, lines 46-49, i.e. Figure 2 illustrates the process by which the image 
converting system 20 generates a three-dimensional model or object from a single two- 
dimensional image and an object contained within the digital image is selected for 
conversion into a three-dimensional object, and the selected object's orientation, or 
natural coordinate system, is approximately determined, wherein the reference 
coordinate system is the natural coordinate system.), 

wherein creating a three dimensional model includes identifying at least one 
boundary of the object and using the identified boundary to associate geometry 
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information with the object (Fig. 2; columns 4-5, lines 65-1 ; column 5, lines 35-38, i.e. 
line creation can be performed automatically by image analysis software, which 
determines edges of objects and creates lines overlapping the determined edges, 
shapes or polygons that define the selected object's sides, and plane indexes are 
determined based on user-designated vertices or vanishing lines at significant features 
of the selected object), 

the geometry information comprising 3-D coordinates describing the position and 
orientation of the object boundary in the reference coordinate system (Fig. 2 (element 
54); column 5, lines 50-55, i.e. once all of the polygons and plane indexes of a selected 
object have been determined, a three-dimensional object, oriented within the selected 
object's three-dimensional coordinate space is determined). 

Therefore it would have been obvious to one of ordinary skill in the art to modify 
the method of Szeliski et al. to include the teachings of Luken and Seago et al. wherein 
directional vectors for each image are used within an environment map that is mapped 
to the sides of an octahedron, as taught by Luken, with the three-dimensional model 
environment map of the visual scene, as taught by Szeliski, because Luken is directed 
to the same problem of using polyhedral environment maps to create and view three 
dimensional images from data representing multiple views of a scene, is in the same 
field of endeavor of image processing systems and expressly suggests that the direction 
vectors provide an efficient system for generating and viewing three-dimensional 
panoramic images based environment maps, and offer an improved level of interactive 
graphical feedback (Luken: column 3, lines 5-8), and wherein the three-dimensional 
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object editing abilities of Seago are used to modify the three-dimensional environment 
map of Szeliski because Seago expressly suggests that this method of object extraction 
produces accurate three-dimensional objects more efficiently than conventional systems 
that try to extract the three-dimensional objects using analytical mathematical 
interpretations and orthogonal image analysis (Seago: column 1 1 , lines 49-55). 

Referring to claim 2, the rationale for claim 1 is incorporated herein, Szeliski et 
al., as modified above, teaches the method of claim 1 wherein the directional vector is 
determined based, at least in part, on instructions identifying elements of the image 
panoramas received from a user (column 8, lines 30-32; column 27, lines 64-66, i.e. a 
user may enter commands and information into the computer through input devices and 
the shape of the model and the embedding of each face into texture space is left up to 
the user). 

Referring to claim 3, the rationale for claim 2 Is incorporated herein, Szeliski et 
al., as modified above, teaches the method of claim 2 wherein the instructions from the 
user identify two or more substantially parallel features in the image panoramas (column 
8, lines 30-32; columns 20-21 , lines 64-6; column 21 , lines 25-34, i.e. a user may enter 
commands and information into the computer through input devices and the global 
alignment method is a feature-based method that relies on first establishing point 
correspondences between overlapping images). 

Referring to claim 5, the rationale for claim 2 is incorporated herein, Szeliski et 
al., as modified above, teaches the method of claim 2 wherein the instructions from the 
user identify a horizon line of at least one image panorama (Figs. 3-4; column 9, lines 
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54-62, i.e. a camera 310 having its optical center fixed at point C (Fig. 3) captures a 
sequence of images lo, li, I2, Is,--- as it pans, the center rays of these images being 
focused on 3D points Po, Pi, P2, Pa,--- at a focal length f from the optical center point C- 
The points P/are defined relative to a fixed 3D world coordinate system Px, Py, Pz 
indicated in the drawings). 

Referring to claim 6, the rationale for claim 2 is incorporated herein, Szelisl^i et 
al., as modified above, teaches the method of claim 2 wherein the instructions comprise 
the identification of two or more areas of the image panoramas, each area containing 
one or more elements and further comprising automatically identifying the two elements 
contained in the two or more areas (Fig. 6; columns 20-21 , lines 49-24, i.e. a feature- 
based point correspondence is established between a pair of images by dividing each 
image into patches and identifying prospective "feature" points within the patches.). 

Referring to claim 8, the rationale for claim 1 is incorporated herein, Szelisl^i et 
al., as modified above, teaches the method of claim 1 but does not specifically teach 
wherein the image panoramas are aligned relative to the reference coordinate system 
such that the directional vector of each panorama is at least substantially parallel to one 
axis of the reference coordinate system. 

Luken teaches wherein the image panoramas are aligned relative to the 
reference coordinate system such that the directional vector of each panorama is at 
least substantially parallel to one axis of the reference coordinate system (Fig. 7 
(element 707); columns 6-7, lines 40-36, i.e. since the six rectangular images are axis 
aligned, then a directional vector that is parallel to one axis must be perpendicular to the 
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other two spatial axes. In other words, in order for a directional vector to intersect one of 
the rectangular images, that vector must be substantially parallel to one axis, which 
requires it to be substantially perpendicular to the others). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Luken and Seago et al. as found in the motivation statement of claim 1 is 
incorporated herein. 

Referring to claim 9, the rationale for claim 1 is incorporated herein, Szeliski et 
al., as modified above, teaches the method of claim 1 but does not specifically teach 

wherein the image panoramas are aligned relative to the reference coordinate system 
such that the directional vector of each panorama is at least substantially orthogonal to 
one axis of the reference coordinate system. 

Luken teaches wherein the image panoramas are aligned relative to the 
reference coordinate system such that the directional vector of each panorama is at 
least substantially orthogonal to one axis of the reference coordinate system (Fig. 6A 
and 7 (element 707); columns 6-7, lines 40-36, i.e. since the six rectangular images are 
axis aligned, then a directional vector that is parallel to one axis must be perpendicular 
to the other two spatial axes. In other words, in order for a directional vector to intersect 
one of the rectangular images, that vector must be substantially parallel to one axis, 
which requires it to be substantially perpendicular to the others). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Luken and Seago et al. as found in the motivation statement of claim 1 is 
incorporated herein. 
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Referring to claim 10, the rationale for claim 1 is incorporated herein, Szeliski et 
al., as modified above, teaches the method of claim 1 wherein the image panoramas 
are aligned according to instructions received from a user (column 27, lines 64-66, i.e. 
the user aligns the image panoramas into texture space.). 

Referring to claim 39, the rationale for claim 1 is incorporated herein, Szeliski et 
a!., as modified above, teaches the method according to claim 1 , wherein the object is a 
room and the at least one boundary of the object is the intersection of a wall of the room 
with the floor (column 27, lines 64-67, i.e. the model can be a cube with six separate 
texture maps for each surface. Using an appropriate environment map would cause the 
top surface to be a ceiling, the bottom surface to be a floor, etc.). 

Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Szeliski et al. In view of Luken and Seago et al., as applied to claim 6 above, and 
further In view of Blank U.S. Patent No. 5469536. 

Referring to claim 7, the rationale for claim 6 is incorporated herein, Szeliski et 
al., as modified above, teaches the method of claim 6 but does not specifically teach 
using edge detection to automatically identify the two elements. 

Blank teaches detecting the edges of an object and separates portions of the 
image that are outside the edge of the object (i.e., the background component) from 
portions of the image that are inside the edge such that the two elements are therefore 
identified as those elements within the edge, and those outside the edge (column 4, 
lines 17-21). 
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Therefore it would have been obvious to one of ordinary skill in the art to modify 
the method of Szeliski et al. to include the teachings of Luken, Seago et al. and Bank 
wherein the edge detection methods, as taught by Blank, are used as an alternative to 
patch-based division, as taught by Szeliski, and as modified by Luken, because it is an 
effective way to divide the image into smaller portions to conquer aligning all aspects of 
an image. 

Claims 11, 13-21, 23, and 36-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Szeliski et al. in view of Seago et al. and Blank. 

Referring to claim 11, Szeliski et al. teaches a computerized method of 
interactively editing objects in a panoramic image, the method comprising: 

receiving an image panorama representing a.visual scene, the image panorama 
having an object and a point source (Figs. 1 , 3 and 36A-38; column 27, Lines 34-48, i.e. 
constructing a complete panoramic mosaic, converting the set of input images and 
associated transforms into one or more images which can be quickly rendered or 
viewed by choosing either a cylindrical or spherical map and converting a rotational 
panorama to a spherical panorama indicates that a plurality of panoramas are being 
received. Figure 6, element 61 0 and column 1 1 , lines 1 6-22 discloses wherein a 
deformation D provides a final correction such that the placement of a fixed object 
relative to the second image's re-warped coordinate system (x", y") is about the same 
as the object's placement relative to the first image's coordinate system (x, y) thus 
indicating that the plurality of image panoramas have an object as claimed). 
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the object occupying a field of view of more tlian 1 80 degrees in tine panorama 
(Figs. 1 , 3, 6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64, i.e. 
one of ordinary skill in the art at the time of invention would recognize that the wall 
objects in figures 33A and 33B and the hotel lobby wall objects are both shown 
occupying a field of view of more than 1 80 degrees since Applicant discloses that 
panoramic images of wall structures indicate objects occupying a field of view of more 
than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph). 

Szeliski et al. does not specifically teach using the computer creating a three 
dimensional model of the visual scene using features of the visual scene and the point 
source, wherein creating a three dimensional model includes identifying at least one 
boundary, of the object and using the identified boundary to associate geometry 
information with the object, the geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary in a reference coordinate 
system; 

using the computer receiving an edit to the object in the panorama; 
using the computer transforming the edit relative to a viewpoint defined by the 
point source; and 

projecting the transformed edit onto the object. 

Seago et al. teaches creating a three dimensional model of the visual scene 
using features of the visual scene and the point source (Fig. 2; column 4, lines 46-49, 
i.e. Figure 2 illustrates the process by which the image converting system 20 generates 
a three-dimensional model or object from a single two-dimensional image and an object 
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contained within the digital image is selected for conversion into a three-dimensional 
object, and the selected object's orientation, or natural coordinate system, is 
approximately determined, wherein the reference coordinate system is the natural 
coordinate system.), 

wherein creating a three dimensional model includes identifying at least one 
boundary, of the object and using the identified boundary to associate geometry 
information with the object (Fig. 2; columns 4-5, lines 65-1 ; column 5, lines 35-38, i.e. 
line creation can be performed automatically by image analysis software, which 
determines edges of objects and creates lines overlapping the determined edges, 
shapes or polygons that define the selected object's sides, and plane indexes are 
determined based on user-designated vertices or vanishing lines at significant features 
of the selected object), 

the geometry information comprising 3-D coordinates describing the position and 
orientation of the object boundary in a reference coordinate system (Fig. 2 (element 54); 
column 5, lines 50-55, i.e. once all of the polygons and plane indexes of a selected 
object have been determined, a three-dimensional object, oriented within the selected 
object's three-dimensional coordinate space is determined). 

Blank teaches using the computer receiving an edit to the object in the panorama 
(Abstract; column 30, lines 18-51 ; column 47, lines 1-64; column 48, lines 26-59, i.e. an 
object is selected for editing and the computer receives edits to trim, sharpen, blur 
blend, size, rotate and distort objects from the user); 
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using tine computer transforming the edit relative to a viewpoint defined by tlie 
point source (Abstract; column 30, lines 18-51 ; column 47, lines 1-64; column 48, lines 
26-59, i.e. an object is selected for editing and the computer receives edits to trim, 
sharpen, blur, blend, size, rotate and distort objects from the user thus indicating that 
the object is being transformed relative to a viewpoint); and 

projecting the transformed edit onto the object (column 30, lines 33-51 , i.e. after 
the user is satisfied with how an object is manipulated a copy of the object is pasted 
onto the background where it then becomes a part of the background image, since 
object edits are performed on the current object displayed in a white box, with other 
objects visible on the screen not being transformed, then the object transform edits are 
being projected onto the current object). 

Therefore it would have been obvious to one of ordinary skill in the art to modify 
the method of Szeliski et al. to include the teachings of Seago et al. and Blank wherein 
the three-dimensional object editing abilities of Seago are used to modify the three- 
dimensional environment map of Szeliski because this method of object extraction 
produces accurate three-dimensional objects more efficiently than conventional systems 
that try to extract the three-dimensional objects using analytical mathematical 
interpretations and orthogonal image analysis (Seago et al.: column 1 1 , lines 49-55) 
and to further include wherein an object is selected for editing and the computer 
receives edits to trim, sharpen, blur, blend, size, rotate and distort objects from the user 
thus indicating that the object is being transformed relative to a viewpoint such that 
object edits are performed on the current object displayed in a white box, with other 
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objects visible on tine screen not being transformed thereby enabling the user to quickly 
and efficiently modify or enhance the appearance of an image to a desired goal (Blank: 
column 6, lines 23-28) 

Referring to claim 13, the rationale for claim 1 1 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 1 1 but does not specifically teach 
receiving an edit to color information associated with the object. 

Blank teaches receiving an edit to color information associated with the object 
(column 47, lines 21-56, i.e. sharpen, blur, fill, tint, blend, gray-scale and adjust color 
are all commands to edit a color thus indicating that these edits are being received). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 

Referring to claim 14, the rationale for claim 1 1 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 1 1 but does not specifically teach 
receiving an edit to alpha information associated with the object. 

Blank teaches this limitation (Fig. 19B; column 47, lines 57-60, i.e. Adjust 
Transparency is a command for adjusting the alpha information of an object). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 



Application/Control Number: 1 0/780,500 Page 1 5 

Art Unit: 2628 

Referring to claim 15, the rationale for claim 1 1 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 1 1 but does not specifically teach 
receiving an edit to depth information associated with the object. 

Blank teaches this limitation (Fig. 19c (element 274); column 2, lines 49-60; 
column 3, lines 55-67; column 5, lines 15-25; column 12, lines 44-62; column 13, lines 
8-15, i.e. the computer blends the object into the selected background at the desired 
layer and X-Y position, wherein objects are moved to the desired depth/layer). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 

Referring to claim 16, the rationale for claim 1 1 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 1 1 but does not specifically teach 
receiving an edit to geometry information associated with the object. 

Blank teaches this limitation (column 47, lines 1 1-20; column 48, lines 26-39 and 
50-59, i.e. the Trim, Size, and Distort commands make changes to the geometry of the 
object). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 

Referring to claim 17, the rationale for claim 1 1 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 1 1 but does not specifically teach 
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providing a user with an interactive drawing tool tliat specifies edits for the object; and 
receiving the edits made by the user using the interactive drawing tool. 

Blank teaches this limitation (column 21 , lines 18-23, i.e. the system acts like a 
highly interactive and very powerful image editing tool). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 Is 
incorporated herein. 

Referring to claim 18, the rationale for claim 17 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 17 wherein the Interactive 
drawing tool Is one of an extrusion tool, a ground plane tool, a depth chisel tool and a 
non-uniform rational B-spline tool. 

Blank teaches this limitation (column 12, lines 44-62; column 13, lines 8-15; 
column 47, lines 11 -20, I.e. the Interactive tool is a depth chisel tool). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 

Referring to claim 19, the rationale for claim 17 Is Incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 17 but does not specifically teach 
wherein the Interactive drawing tool specifies a selected value for depth for the object. 

Blank teaches this limitation (column 12, lines 44-62; column 13, lines 8-15, i.e. 
the user selects the desired layer/depth of a selected object). 
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The rationale for combining Szeliski et a!., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 

Referring to claim 20, the rationale for claim 17 is incorporated herein, Szeliski 
et a!., as modified above, teaches the method of claim 17 but does not specifically teach 
wherein the interactive drawing tool incrementally adds to the depth for the object. 

Blank teaches this limitation (column 12, lines 44-62; column 13, lines 8-15, i.e. 
objects can be chosen to have a higher priority and hence a higher numbered layer and 
Z coordinate such that their priority may be incremented to add to their depth). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 

Referring to claim 21, the rationale for claim 17 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 17 but does not specifically teach 
wherein the interactive drawing tool incrementally subtracts from the depth for the 
object. 

Blank teaches this limitation (column 1 2, lines 44-62; column 1 3, lines 8-1 5, i.e. 
objects can be chosen to have a lower priority and hence a lower numbered layer and Z 
coordinate such that their priority may be decremented to subtract from their depth). 

The rationale for combining Szeliski et al., as modified above, with the teachings 
of Seago et al. and Blank as found in the motivation statement of claim 1 1 is 
incorporated herein. 
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The rationale of claim 17 is applied to claim 23. 

Referring to claim 36, claim 36 recites elements that are similar in scope to 
claims 1 1 and 1 7, and therefore the rationale for the rejection of claims 1 1 and 1 7 are 
incorporated herein. 

Referring to claim 37, claim 37 recites all of the elements of claims 1 8 and 36, 
and therefore the rationale for the rejection of claims 18 and 36 are incorporated herein. 

Claims 22, 24-28, 32-35, 38 and 40-41 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Szeliski et al. in view of Seago et al. 

Referring to claim 22, Szeliski et al. teaches a computerized method for 
projecting texture information onto an object within an image panorama, the method 
comprising: 

using the computer receiving instructions from a user identifying a three- 
dimensional geometric surface within an image panorama, the image panorama 
containing an object having one or more textures (Figs. 1 , 3 and 36A-38; column 27, 
lines 34-41 ; columns 27-28, lines 60-12, i.e. the shape of the model and the embedding 
of each face are left up to the user such that the user can choose something as simple 
as a cube with six separate texture maps or a spherical texture map. Figure 6, element 
61 0 and column 1 1 , lines 1 6-22 discloses wherein a deformation D provides a final 
correction such that the placement of a fixed object relative to the second image's re- 
warped coordinate system (x", y") is about the same as the object's placement relative 
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to the first image's coordinate system (x, y) thus indicating that the plurality of image 
panoramas have an object as claimed), 

the object occupying a field of view of more than 1 80 degrees in the panorama 
(Figs. 1 , 3, 6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64, i.e. 
one of ordinary skill in the art at the time of invention would recognize that the wall 
objects in figures 33A and 33B and the hotel lobby wall objects are both shown 
occupying a field of view of more than 1 80 degrees since Applicant discloses that 
panoramic images of wall structures indicate objects occupying a field of view of more 
than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph); 

determining a directional vector from the three-dimensional geometric surface 
(Fig. 15; column 22, lines 4-31 , i.e. figure 15 shows the adjustment of the bundle of rays 
Xjk so that they converge to xy. Letting the ray direction in the final composited image 
mosaic be a unit vector p, and its corresponding ray direction in the kth frame be py/f-R/c' 
V/f'' x;/f, indicates a directional vector from the surface as claimed); 

using the computer creating a geometric model of the image panorama based at 
least in part on the three-dimensional geometric surface and the directional vector (Figs. 
17-18 and 20; column 7, lines 29-33; column 22, lines 2-31 and 46-67; column 23, lines 
10-12; column 24, lines 30-61 , i.e. the direction vectors are used to align the images of 
the panoramic image such that global alignment of the image panoramas is being 
performed using patch-based alignment); and 
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applying the one or more textures to the object in the image panorama based on 
the geometric model (Fig. 2B; column 28, lines 13-15, i.e. efficiently computing texture 
map color values for any geometry and choice of texture map coordinates). 

Szeliski et al. does not specifically teach wherein creating a geometric model 
includes identifying at least one boundary of the object and using the identified 
boundary to associate geometry information with the object, the geometry information 
comprising 3-D coordinates describing the position and orientation of the object 
boundary in a reference coordinate system. 

Seago et al. teaches creating a three dimensional model of the visual scene (Fig. 
2; column 4, lines 46-49, i.e. Figure 2 illustrates the process by which the image 
converting system 20 generates a three-dimensional model or object from a single two- 
dimensional image and an object contained within the digital image is selected for 
conversion into a three-dimensional object, and the selected object's orientation, or 
natural coordinate system, is approximately determined, wherein the reference 
coordinate system is the natural coordinate system.), 

wherein creating a three dimensional model includes identifying at least one 
boundary of the object and using the identified boundary to associate geometry 
information with the object (Fig. 2; columns 4-5, lines 65-1 ; column 5, lines 35-38, i.e. 
line creation can be performed automatically by image analysis software, which 
determines edges of objects and creates lines overlapping the determined edges, 
shapes or polygons that define the selected object's sides, and plane indexes are 



Application/Control Number: 10/780,500 Page 21 

Art Unit: 2628 

determined based on user-designated vertices or vanishing lines at significant features 
of the selected object), 

the geometry information comprising 3-D coordinates describing the position and 
orientation of the object boundary in the reference coordinate system (Fig. 2 (element 
54); column 5, lines 50-55, i.e. once all of the polygons and plane indexes of a selected 
object have been determined, a three-dimensional object, oriented within the selected 
object's three-dimensional coordinate space is determined). 

Therefore it would have been obvious to one of ordinary skill in the art to modify 
the method of Szeliski et al. to include the teachings of Seago et al. wherein the three- 
dimensional object editing abilities of Seago are used to modify the three-dimensional 
environment map of Szeliski because Seago expressly suggests that this method of 
object extraction produces accurate three-dimensional objects more efficiently than 
conventional systems that try to extract the three-dimensional objects using analytical 
mathematical interpretations and orthogonal image analysis (Seago: column 1 1 , lines 
49-55). 

Referring to claim 24, the rationale for claim 22 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 22 wherein the three-dimensional 
geometric surface is one of a floor, a wall, or a ceiling (column 27, lines 60-67, i.e. the 
shape of the model and the embedding of each face into texture space can be a simple 
cube with six separate texture maps for each surface. Using an appropriate 
environment map would result in the top surface being a ceiling, the bottom surface 
being a floor and the side surfaces being a wall, see figures33A-33D). 
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Referring to claim 25, the rationale for claim 22 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 22 wherein the directional vector 
is orthogonal to the planar surface (Fig. 15, column 22, lines 4-6, i.e. the ray direction 
going through the 'f" feature point located at xjk, yjk in the k''' frame indicates that the 
directional vector Is orthogonal as claimed). 

Referring to claim 26, the rationale for claim 22 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 22 wherein the geometric model 
comprises depth information (Figs. 27 and 30; column 28, lines 19-33, i.e. the object 
model Is a triangulated surface where each vertex is tagged with its 3D (X,, Y, Z) 
coordinates). 

Referring to claim 27, the rationale for claim 22 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 22 wherein the texture 
Information comprises color Information (column 28, lines 13-18, I.e. texture map color 
values are computed for any geometry and choice of texture map coordinates). 

Referring to claim 28, the rationale for claim 22 is incorporated herein, Szeliski 
et al., as modified above, teaches the method of claim 22 wherein the texture 
information comprises luminance information (Fig. 2B; column 5, lines 27-30; column 
1 1 , lines 57; column 29, lines 30-40 and 60-65, i.e. one of ordinary skill In the art at the 
time of Invention would recognize that intensity is equivalent to luminance). 

Referring to claim 32, Szeliski et al. teaches a system for creating a three 
dimensional model from a plurality of image panoramas, the system comprising: 
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means for receiving tine image panoramas representing a visual scene liaving an 
object (Figs. 1 , 3 and 36A-38; column 27, lines 34-41 ; columns 27-28, lines 60-12, i.e. 
the shape of the model and the embedding of each face are left up to the user such that 
the user can choose something as simple as a cube with six separate texture maps or a 
spherical texture map. Figure 6, element 610 and column 1 1 , lines 16-22 discloses 
wherein a deformation D provides a final correction such that the placement of a fixed 
object relative to the second image's re-warped coordinate system (x", y") is about the 
same as the object's placement relative to the first image's coordinate system (x, y) thus 
indicating that the plurality of image panoramas have an object as claimed); 

the object occupying a field of view of more than 1 80 degrees in the panoramas 
(Figs. 1 , 3, 6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64, i.e. 
one of ordinary skill in the art at the time of invention would recognize that the wall 
objects in figures 33A and 33B and the hotel lobby wall objects are both shown 
occupying a field of view of more than 180 degrees since Applicant discloses that 
panoramic images of wall structures indicate objects occupying a field of view of more 
than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph), 

means for allowing a user to interact with the system to determine a directional 
vector for each image panorama (Fig. 1 5; column 22, lines 4-31 , i.e. figure 1 5 shows the 
adjustment of the bundle of rays xjk so that they converge to xy. Letting the ray direction 
in the final composited image mosaic be a unit vector p, and its corresponding ray 
direction in the kth frame be py^-R/c'V^'^xy/c, indicates a directional vector from the 
surface as claimed); 
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means for aligning tlie image panoramas relative to each other (Figs. 17-18 and 
20; column 7, lines 29-33; column 22, lines 2-31 and 46-67; column 23, lines 10-12; 
column 24, lines 30-61 , i.e. the direction vectors are used to align the images of the 
panoramic image such that global alignment of the image panoramas is being 
performed using patch-based alignment); and 

means for creating a three dimensional model from the aligned (Figs. 1 7-1 8 and 
20; column 7, lines 29-33; column 22, lines 2-31 and 46-67; column 23, lines 10-12; 
column 24, lines 30-61 , i.e. the direction vectors are used to align the images of the 
panoramic image such that global alignment of the image panoramas is being 
performed using patch-based alignment). 

Szeliski et al. does not specifically teach wherein creating a geometric model 
includes identifying at least one boundary of the object and using the identified 
boundary to associate geometry information with the object, the geometry information 
comprising 3-D coordinates describing the position and orientation of the object 
boundary in a reference coordinate system. 

Seago et al. teaches creating a three dimensional model of the visual scene (Fig. 
2; column 4, lines 46-49, i.e. Figure 2 illustrates the process by which the image 
converting system 20 generates a three-dimensional model or object from a single two- 
dimensional image and an object contained within the digital image is selected for 
conversion into a three-dimensional object, and the selected object's orientation, or 
natural coordinate system, is approximately determined, wherein the reference 
coordinate system is the natural coordinate system.). 
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wherein creating a tliree dimensional model includes identifying at least one 
boundary of the object and using the identified boundary to associate geometry 
information with the object (Fig. 2; columns 4-5, lines 65-1 ; column 5, lines 35-38, i.e. 
line creation can be performed automatically by image analysis software, which 
determines edges of objects and creates lines overlapping the determined edges, 
shapes or polygons that define the selected object's sides, and plane indexes are 
determined based on user-designated vertices or vanishing lines at significant features 
of the selected object), 

the geometry information comprising 3-D coordinates describing the position and 
orientation of the object boundary in the reference coordinate system (Fig. 2 (element 
54); column 5, lines 50-55, i.e. once all of the polygons and plane indexes of a selected 
object have been determined, a three-dimensional object, oriented within the selected 
object's three-dimensional coordinate space is determined). 

Therefore it would have been obvious to one of ordinary skill in the art to modify 
the method of Szeliski et al. to include the teachings of Seago et al. wherein the three- 
dimensional object editing abilities of Seago are used to modify the three-dimensional 
environment map of Szeliski because Seago expressly suggests that this method of 
object extraction produces accurate three-dimensional objects more efficiently than 
conventional systems that try to extract the three-dimensional objects using analytical 
mathematical interpretations and orthogonal image analysis (Seago: column 11, lines 
49-55). 
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Referring to claim 33, the rationale for claim 32 is incorporated herein, Szeliski 
et al., as modified above, teaches the system of claim 32, wherein the input image 
panoramas comprise two-dimensional images (Figs. 3-5; columns 9-10, lines 56-1, i.e. 
a camera captures 2D still images that are registered so that a panorama may be 

constructed). 

Referring to claim 34, the rationale for claim 32 is incorporated herein, Szeliski 
et al., as modified above, teaches the system of claim 32, wherein the input image 
panoramas comprise three-dimensional images including geometry information (Figs. 3- 
4 and 6; column 9, lines 56-62, i.e. the camera captures a sequence of 2D still images 
(lo, h, I2, la-) as it pans, the center rays of these images being focused on 3D points (Po, 
Pi, P2, P3...) at a focal length f from the optical center point C. The points Pi are defined 
relative to a fixed 3D world coordinate system (Px, Py, Pz)- Since the three-dimensional 
images correspond to the two-dimensional images which include depth information in 
the form of focal length then the geometry information is the inverted V shape shown in 
both figures 4 and 6). 

Referring to claim 35, the rationale for claim 32 is incorporated herein, Szeliski 
et al., as modified above, teaches the system of claim 32, wherein the image 
panoramas are aligned according to instructions received from a user (Figs. 2B, 17-18 
and 20; column 22, lines 46-67; column 23, lines 1 0-12; column 24, lines 30-61 ; column 
27, lines 64-66, i.e. the user aligns the image panoramas into texture space.). 

Referring to claim 38, Szeliski et al. teaches a computerized method for 
creating a three dimensional model from an image panorama, the method comprising: 
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receiving an image panorama representing a visual scene and liaving an object 
(Figs. 1 , 3 and 36A-38; column 27, Lines 34-48, i.e. constructing a complete panoramic 
mosaic, converting the set of input images and associated transforms into one or more 
images which can be quickly rendered or viewed by choosing either a cylindrical or 
spherical map and converting a rotational panorama to a spherical panorama indicates 
that a plurality of panoramas are being received. Figure 6, element 61 0 and column 1 1 , 
lines 1 6-22 discloses wherein a deformation D provides a final correction such that the 
placement of a fixed object relative to the second image's re-warped coordinate system 
(x", y") is about the same as the object's placement relative to the first image's 
coordinate system (x, y) thus indicating receiving an image panorama having an object 
as claimed), 

the object occupying a field of view of more than 1 80 degrees in the panorama 
(Figs. 1 , 3, 6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64, i.e. 
one of ordinary skill in the art at the time of invention would recognize that the wall 
objects in figures 33A and 33B and the hotel lobby wall objects are both shown 
occupying a field of view of more than 1 80 degrees since Applicant discloses that 
panoramic images of wall structures indicate objects occupying a field of view of more 
than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"*^ paragraph); and 

creating a three dimensional model of the visual scene using a computer from 
the image panorama using the reference coordinate system (Figs. 17-18 and 20; 
column 7, lines 29-33; column 22, lines 2-31 and 46-67; column 23, lines 10-12; column 
24, lines 30-61 , i.e. the direction vectors are used to align the images of the panoramic 
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image sucli tliat global alignment of the image panoramas is being performed using 
patch-based alignment). 

Szeliski et al. does not specifically teach wherein creating a geometric model 
includes identifying at least one boundary of the object and using the identified 
boundary to associate geometry information with the object, the geometry information 
comprising 3-D coordinates describing the position and orientation of the object 
boundary in a reference coordinate system. 

Seago et al. teaches creating a three dimensional model of the visual scene (Fig. 
2; column 4, lines 46-49, i.e. Figure 2 illustrates the process by which the image 
converting system 20 generates a three-dimensional model or object from a single two- 
dimensional image and an object contained within the digital image is selected for 
conversion into a three-dimensional object, and the selected object's orientation, or 
natural coordinate system, is approximately determined, wherein the reference 
coordinate system is the natural coordinate system.), 

wherein creating a three dimensional model includes identifying at least one 
boundary of the object and using the identified boundary to associate geometry 
information with the object (Fig. 2; columns 4-5, lines 65-1 ; column 5, lines 35-38, i.e. 
line creation can be performed automatically by image analysis software, which 
determines edges of objects and creates lines overlapping the determined edges, 
shapes or polygons that define the selected object's sides, and plane indexes are 
determined based on user-designated vertices or vanishing lines at significant features 
of the selected object). 
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the geometry information comprising 3-D coordinates describing tine position and 
orientation of the object boundary in the reference coordinate system (Fig. 2 (element 
54); column 5, lines 50-55, i.e. once all of the polygons and plane indexes of a selected 
object have been determined, a three-dimensional object, oriented within the selected 
object's three-dimensional coordinate space is determined). 

Therefore it would have been obvious to one of ordinary skill in the art to modify 
the method of Szeliski et al. to include the teachings of Seago et al. wherein the three- 
dimensional object editing abilities of Seago are used to modify the three-dimensional 
environment map of Szeliski because Seago expressly suggests that this method of 
object extraction produces accurate three-dimensional objects more efficiently than 
conventional systems that try to extract the three-dimensional objects using analytical 
mathematical interpretations and orthogonal image analysis (Seago: column 11, lines 
49-55). 

Referring to claim 40, the rationale for claim 22 is incorporated herein, Szeliski 
et al., as modified above, teaches the method according to claim 22, wherein the object 
is a room and the at least one boundary of the object is the intersection of a wall of the 
room with the floor (column 27, lines 64-67, i.e. the model can be a cube with six 
separate texture maps for each surface. Using an appropriate environment map would 
cause the top surface to be a ceiling, the bottom surface to be a floor, etc.). 

Referring to claim 41, claim 41 recites all of the elements of claims 40 and 38 
and therefore the rationale for the rejection of claims 40 and 38 are incorporated herein. 
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Response to Arguments 

Applicant's arguments filed 12/21/2010 have been fully considered but they are 
not persuasive. 

Applicant first argues, with respect to ciaims 22-28, 32-35, 38 and 40-41, 

that neither Szeliski nor Seago teaches creating a geometric model of an object within 
an image panorama including identifying at least one boundary of the object where the 
object occupies a field of view in the panorama of grater than 1 80 degrees as required 
by each of the claims. 

Examiner respectfully submits that Szeliski et al. teaches receiving panoramic 
images containing an object occupying a field of view of more than 180 degrees, see 
Figs. 1,3,6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64. One 
of ordinary skill in the art at the time of invention would recognize that the Szeliski 
reference teaches an object occupying such a field of view because the wall objects in 
figures 33A and 33B and the hotel lobby wall and floor objects are both shown 
occupying a field of view of more than 180 degrees since Applicant themselves 
disclosed that panoramic images of wall structures indicate objects occupying a field of 
view of more than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph. 
Szeliski et al. further teaches the use of traditional texture-mapped models, i.e. 
environment maps, wherein the shape of the model may be chosen as a simple cube 
with six separate texture maps indicating an environment map whose sides occupy a 
field of view of more than 180 degrees, see column 27, lines 60-67. Szeliski et al. also 
teaches an object model that is a collection of triangles and vertices wherein each 
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vertex is tagged witli its 3D (X, Y, and Z) coordinates and 2D (u, v) coordinates whose 
faces may be assigned to different texture maps. Thus Szeliski discloses the ability to 
generate a 3D model of an object in an image wherein the 3D model may occupy a field 
of view of more than 180 degrees. Szeliski does not specifically teach wherein creating 
the geometric model includes identifying at least one boundary of the object and using 
the identified boundary to associate geometry information with the object, the geometry 
information comprising 3-D coordinates describing the position and orientation of the 
object boundary in a reference coordinate system. Seago is depended upon to teach 
wherein creating a geometric model includes identifying at least one boundary of the 
object and using the identified boundary to associate geometry information with the 
object, the geometry information comprising 3-D coordinates describing the position and 
orientation of the object boundary in a reference coordinate system, see Fig. 2; column 
4, lines 46-49, columns 4-5, lines 65-1 ; column 5, lines 35-38 and 50-55. One of 
ordinary skill in the art at the time of invention would be able to apply the method of 
Seago, which takes 2-D planar images, determines vanishing points to derive a 3D 
coordinate system, and models the objects within the images, to the teachings of 
Szeliski which also includes the use of 2D perspective images and 3D models of objects 
within the images in order to improve Szeliski because Seago's method of object 
extraction produces accurate three-dimensional objects more efficiently than 
conventional systems that try to extract the three-dimensional objects using analytical 
mathematical interpretations and orthogonal image analysis (Seago et al.: column 11, 
lines 49-55). 
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Applicant then argues, with respect to claim 22, that if tlie combination of 
Szeliski and Seago teaches limitation (b) of claim 22, Seago '900 must supply the 
teaching of modeling the object using an identified boundary of the object where the 
object occupies the field of view of more than 1 80 degrees in the panorama as required 
by limitation (a) of claim 22. However, Seago's method fails when the object to be 
modeled occupies a field of view greater than 180 degrees in the input image(s). 

Examiner respectfully submits that Seago teaches wherein a sufficient set of 
features may include any set of matching or conjugate vertices, vanishing lines, or 
planes between the multiple images, see column 7, lines 35-41 , and further goes on to 
disclose that the relative orientation of a view is how the vertices, vanishing lines, or 
planes it sees correspond to those other views. For example, if two views of the same 
object differ by 180 degrees, the left vanishing lines and planes in one view are the 
right vanishing lines and planes in the other view, see column 7, lines 42-49. One of 
ordinary skill in the art would recognize that this disclosure indicates that the method of 
Seago would indeed work for objects that cover a field of view greater than 1 80 degrees 
(as disclosed in Szeliski) and therefore the combination of primary reference Szeliski et 
al. with secondary reference Seago et al. teaches all of the elements of claims 22-28, 
32-35, 38 and 40-41 as claimed. Szeliski et al. teaches receiving panoramic images 
containing an object occupying a field of view of more than 180 degrees, see Figs. 1 , 3, 
6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64. One of ordinary 
skill in the art at the time of invention would recognize that the Szeliski reference 
teaches an object occupying such a field of view because the wall objects in figures 33A 
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and 33B and the hotel lobby wall and floor objects are both shown occupying a field of 
view of more than 1 80 degrees since Applicant themselves disclosed that panoramic 
images of wall structures indicate objects occupying a field of view of more than 180 
degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph. Szeliski et al. further 
teaches the use of traditional texture- mapped models, i.e. environment maps, wherein 
the shape of the model may be chosen as a simple cube with six separate texture maps 
indicating an environment map whose sides occupy a field of view of more than 180 
degrees, see column 27, lines 60-67. Szeliski et al. also teaches an object model that is 
a collection of triangles and vertices wherein each vertex is tagged with its 3D (X, Y, 
and Z) coordinates and 2D (u, v) coordinates whose faces may be assigned to different 
texture maps. Thus Szeliski discloses the ability to generate a 3D model of an object in 
an image wherein the 3D model may occupy a field of view of more than 1 80 degrees. 

Applicant then argues, with respect to claim 22, that the input to Seago's method 
is a 2-D perspective image or a group of 2-D perspective images displaying the object 
to be modeled. However, when the object to be modeled requires a field of view of more 
than 180 degrees, the entire object will not fit onto a single 2D perspective image and 
Seago's method requires that a separate 3D model be made of the object from each of 
the 2D perspective images. When each 2D image does not contain the entire object, 
Seago's method fails because the required 3D models of the object cannot be created. 

Examiner respectfully submits that Seago teaches wherein a single three- 
dimensional object is created from two or more three-dimensional objects created 
from a plurality of two-dimensional images of a single object, see column 7, lines 29-65, 
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and also teaches wherein a single three-dimensional object is created from a single 
two-dimensional image, see column 4, lines 34-65, thus Seago's method does not fail 
as argued. Szeliski teaches generating a single panoramic Image from a plurality of 
two-dimensional perspective images, see columns 9-10, lines 53-6 and column 31 , lines 
44-64. Thus, the combination of primary reference Szeliski with secondary reference 
Seago teaches all of the elements of claim 22. 

Applicant then uses figures 1-6 to indicate that because the object requires a 
field of view of more than 1 80 degrees, the object will not fit on a single 2D perspective 
image and that the curved wall object in figure 1 wraps around the camera object but to 
create a mosaic similar to Szeliski the user must take multiple overlapping photos to 
span the wall's extent, which is greater than 180 degrees such that the mosaiced image 
to be used comprises three 2D images that have sufficient overlap to register the 
images. But that the final mosaic created with the three initial images cannot be 
displayed as a single 2D perspective image that preserves the features that are used to 
register or model the scene and objects within the scene. 

In response to applicant's argument that the references fail to show certain 
features of applicant's invention, it is noted that the features upon which applicant relies 
(i.e., the final mosaic created with the three initial images cannot be displayed as a 
single 2D perspective image that preserves the features that are used to register or 
model the scene and objects within the scene, see Remarks, page 1 9 of 23) are not 
recited in the rejected claim(s). Although the claims are interpreted in light of the 
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specification, limitations from the specification are not read into the claims. See In re 
Van Geuns, 988 F.2d 1 181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

Examiner respectfully submits that Szeliski et al. teaches receiving panoramic 
images containing an object occupying a field of view of more than 180 degrees, see 
Figs. 1,3,6, 33A-B, and 36A-38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64. One 
of ordinary skill in the art at the time of invention would recognize that the Szeliski 
reference teaches an object occupying such a field of view because the wall objects in 
figures 33A and 33B and the hotel lobby wall and floor objects are both shown 
occupying a field of view of more than 180 degrees since Applicant themselves 
disclosed that panoramic images of wall structures indicate objects occupying a field of 
view of more than 180 degrees, see Remarks, filed 2/28/2008, page 11,2"'' paragraph. 
Szeliski et al. further teaches the use of traditional texture-mapped models, i.e. 
environment maps, wherein the shape of the model may be chosen as a simple cube 
with six separate texture maps indicating an environment map whose sides occupy a 
field of view of more than 180 degrees, see column 27, lines 60-67. Szeliski et al. also 
teaches an object model that is a collection of triangles and vertices wherein each 
vertex is tagged with its 3D (X, Y, and Z) coordinates and 2D (u, v) coordinates whose 
faces may be assigned to different texture maps. Thus Szeliski discloses the ability to 
generate a 3D model of an object in an image wherein the 3D model may occupy a field 
of view of more than 1 80 degrees. Seago teaches wherein a sufficient set of features 
may include any set of matching or conjugate vertices, vanishing lines, or planes 
between the multiple images, see column 7, lines 35-41 , and further goes on to disclose 
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that the relative orientation of a view is how the vertices, vanishing lines, or planes it 
sees correspond to those other views. For example, if two views of the same object 
differ by 180 degrees, the left vanishing lines and planes in one view are the right 
vanishing lines and planes in the other view, see column 7, lines 42-49. One of ordinary 
skill in the art would recognize that this disclosure indicates that the method of Seago 
would indeed work for objects that cover a field of view greater than 1 80 degrees and 
therefore the combination of primary reference Szeliski et al. with secondary reference 
Seago et al. teaches all of the elements of claims 22-28, 32-35, 38 and 40-41 as 
claimed. 

Applicant then argues that the warping needed to force objects with extents 
greater than 1 80 degrees onto a single 2D image destroys the perspective view of these 
objects and that such warping renders the single image useless for Seago's method, 
which demands perspective images of objects as input. 

Examiner respectfully submits that one of ordinary skill in the art at the time of 
invention would be able to apply the method of Seago, which takes 2-D planar images, 
determines vanishing points to derive a 3D coordinate system, and models the objects 
within the images, to the teachings of Szeliski which also includes the use of 2D 
perspective images and 3D models of objects within the images in order to improve 
Szeliski because Seago's method of object extraction produces accurate three- 
dimensional objects more efficiently than conventional systems that try to extract the 
three-dimensional objects using analytical mathematical interpretations and orthogonal 
image analysis (Seago et al.: column 1 1 , lines 49-55). 
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According to KSR Int'l Co. v. Teleflex, Inc., 127 S. Ct. 1727, 1739 (2007) at 1396, 
"if a technique has been used to improve one device, and a person of ordinary skill in 
the art would recognize that it would improve similar devices in the same way, using the 
technique is obvious unless its actual application is beyond his or her skill." As 
established above, both Seago and Szeliski use 2D perspective images and one of 
ordinary skill would recognize how to create object models from the 2D perspective 
images of Szeliski using the method of Seago. Examiner notes that the perspective 
information required to perform the method of Seago is not eliminated in Szeliski, 
regardless of whether the objects in the images occupy a field of view greater than 1 80 
degrees. Therefore, Examiner maintains her position that the combination of Szeliski 
and Seago would have been obvious and teaches limitations for which they are cited. 

Applicant then argues, with respect to claims 22-28, 32-35, 38 and 40-41 , that 
"Clearly, if any of the object's three orthogonal sides cannot be located in each image, 
steps 42 and 44 cannot be performed. Using the simplest example of a wall that does 
not fit on a single 2D image, consider the process of geometrically modeling the wall, 
which is taken as two overlapping images, A and B." and cites to figures 8-12 for 
support, stating that because no features are visible for portions of the wall object in the 
various images then the slope and position of the lines 3 and 4 cannot be determined 
unambiguously. 

Examiner respectfully submits that Seago does not impose a 180 degree limit as 
argued, Seago teaches wherein a sufficient set of features may include any set of 
matching or conjugate vertices, vanishing lines, or planes between the multiple images. 
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see column 7, lines 35-41 , and further goes on to disclose that the relative orientation of 
a view is how the vertices, vanishing lines, or planes it sees correspond to those other 
views. For example, if two views of the same object differ by 180 degrees, the left 
vanishing lines and planes in one view are the right vanishing lines and planes in the 
other view, see column 7, lines 42-49. One of ordinary skill in the art would recognize 
that this disclosure indicates that the method of Seago would indeed work for objects 
that cover a field of view greater than 180 degrees and therefore the combination of 
primary reference Szeliski et al. with secondary reference Seago et al. teaches all of the 
elements of claims 22-28, 32-35, 38 and 40-41 as claimed. 

One of ordinary skill in the art at the time of invention would be able to apply the 
method of Seago, which takes 2-D planar images, determines vanishing points to derive 
a 3D coordinate system, and models the objects within the images, to the teachings of 
Szeliski which also includes the use of 2D perspective images and 3D models of objects 
within the images in order to improve Szeliski because Seago's method of object 
extraction produces accurate three-dimensional objects more efficiently than 
conventional systems that try to extract the three-dimensional objects using analytical 
mathematical interpretations and orthogonal image analysis (Seago et al.: column 11, 
lines 49-55). 

According to KSR Int'l Co. v. Teleflex, Inc., 127 S. Ct. 1727, 1739 (2007) at 1396, 
"if a technique has been used to improve one device, and a person of ordinary skill in 
the art would recognize that it would improve similar devices in the same way, using the 
technique is obvious unless its actual application is beyond his or her skill." As 
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established above, both Seago and Szeliski use 2D perspective images and one of 
ordinary skill would recognize how to create object models from the 2D perspective 
images of Szeliski using the method of Seago. Examiner notes that the perspective 
information required to perform the method of Seago is not eliminated in Szeliski, 
regardless of whether the objects in the images occupy a field of view greater than 180 
degrees. Therefore, Examiner maintains her position that the combination of Szeliski 
and Seago would have been obvious and teaches limitations for which they are cited. 

Applicant then argues, with respect to claims 22-28, 32-35, 38 and 40-41 , that 
"Steps 42 and 44 in fig. 2 of Seago '900 cannot be performed accurately for image B for 
the rightmost orthogonal side of the wall object. Thus, Seago's method will fail to 
produce a 3D model of the wall from image B. Thus, Seago's method will fail because 
separate 3D models cannot be created from each of the multiple 2D perspective images 
needed to show an object with extent > 180 degrees." 

Examiner respectfully submits that Seago teaches wherein a sufficient set of 
features may include any set of matching or conjugate vertices, vanishing lines, or 
planes between the multiple images, see column 7, lines 35-41 , and further goes on to 
disclose that the relative orientation of a view is how the vertices, vanishing lines, or 
planes it sees correspond to those other views. For example, if two views of tfie same 
object differ by 180 degrees, the left vanishing lines and planes in one view are the 
right vanishing lines and planes in the other view, see column 7, lines 42-49. One of 
ordinary skill in the art would recognize that this disclosure indicates that the method of 
Seago would indeed work for objects that cover a field of view greater than 1 80 degrees 
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and therefore the combination of primary reference Szeliski et al. with secondary 
reference Seago et al. teaches all of the elements of claims 22-28, 32-35, 38 and 40-41 
as claimed. Szeliski et al. teaches receiving panoramic images containing an object 
occupying a field of view of more than 1 80 degrees, see Figs. 1,3,6, 33A-B, and 36A- 
38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64. One of ordinary skill in the art at the 
time of invention would recognize that the Szeliski reference teaches an object 
occupying such a field of view because the wall objects in figures 33A and 33B and the 
hotel lobby wall and floor objects are both shown occupying a field of view of more than 
1 80 degrees since Applicant themselves disclosed that panoramic images of wall 
structures indicate objects occupying a field of view of more than 180 degrees, see 
Remarks, filed 2/28/2008, page 11,2"'' paragraph. Szeliski et al. further teaches the use 
of traditional texture-mapped models, i.e. environment maps, wherein the shape of the 
model may be chosen as a simple cube with six separate texture maps indicating an 
environment map whose sides occupy a field of view of more than 180 degrees, see 
column 27, lines 60-67. Szeliski et al. also teaches an object model that is a collection 
of triangles and vertices wherein each vertex is tagged with its 3D (X, Y, and Z) 
coordinates and 2D (u, v) coordinates whose faces may be assigned to different texture 
maps. Thus Szeliski discloses the ability to generate a 3D model of an object in an 
image wherein the 3D model may occupy a field of view of more than 180 degrees. 

Applicant then argues, with respect to ciaims 22-28, 32-35, 38 and 40-41, 
that "To be sure, Szeliski '774's method can be used to produce a single 2D image that 
has four vertices visible to model in the case of the wall shown in figs. 8-1 2, since the 
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wall's extent is not greater than 180 degrees. But for objects that require more than a 
180 degree field of view, the necessary vertices (and edges/planes) cannot be projected 
onto a single 2D perspective image to satisfy Seago '900's method. (See figs. 1-7 and 
accompanying discussion above.) If a mosaic/panorama with an object extends beyond 
1 80 degrees, the Image needs to be warped/remapped to fit into a single 2D image. 
Warping or remapping features does not work for Seago's method because Seago's 
method relies on perspective, e.g., straight lines on objects are no longer straight, etc." 

Examiner respectfully submits that Seago teaches wherein a sufficient set of 
features may include any set of matching or conjugate vertices, vanishing lines, or 
planes between the multiple images, see column 7, lines 35-41 , and further goes on to 
disclose that the relative orientation of a view is how the vertices, vanishing lines, or 
planes it sees correspond to those other views. For example, if two views of the same 
object differ by 180 degrees, the left vanishing lines and planes in one view are the 
right vanishing lines and planes in the other view, see column 7, lines 42-49. One of 
ordinary skill in the art would recognize that this disclosure indicates that the method of 
Seago would indeed work for objects that cover a field of view greater than 1 80 degrees 
and therefore the combination of primary reference Szeliski et al. with secondary 
reference Seago et al. teaches all of the elements of claims 22-28, 32-35, 38 and 40-41 
as claimed. Szeliski et al. teaches receiving panoramic images containing an object 
occupying a field of view of more than 1 80 degrees, see Figs. 1,3,6, 33A-B, and 36A- 
38; column 1 1 , lines 1 6-21 ; column 31 , lines 51 -64. One of ordinary skill in the art at the 
time of invention would recognize that the Szeliski reference teaches an object 
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occupying sucli a field of view because tfie wall objects in figures 33A and 33B and tlie 
hotel lobby wall and floor objects are both shown occupying a field of view of more than 
180 degrees since Applicant themselves disclosed that panoramic images of wall 
structures indicate objects occupying a field of view of more than 180 degrees, see 
Remarks, filed 2/28/2008, page 1 1 , 2"^^ paragraph. SzeliskI et al. further teaches the use 
of traditional texture-mapped models, i.e. environment maps, wherein the shape of the 
model may be chosen as a simple cube with six separate texture maps indicating an 
environment map whose sides occupy a field of view of more than 1 80 degrees, see 
column 27, lines 60-67. It is noted that the inputs to the image panorama of Szeliski et 
al. is a series of two-dimensional perspective view images that are registered to 
construct the panoramic mosaic, see columns 9-10, lines 510. Szeliski et al. also 
teaches an object model that is a collection of triangles and vertices wherein each 
vertex is tagged with its 3D (X, Y, and Z) coordinates and 2D (u, v) coordinates whose 
faces may be assigned to different texture maps. Thus Szeliski discloses the ability to 
generate a 3D model of an object in an image wherein the 3D model may occupy a field 
of view of more than 1 80 degrees. 

Applicant then argues, with respect to claims 11, 13-21, 23 and 36-37 over 
Szeliski, Seago and Blank, claims 1-3, 5-6, 8-10, and 39 over Szeliski, Seago and 
Luken, and claim 7 over Szeliski, Seago, Luken and Blank, that the rejections of 
these claims rely on Szeliski and Seago for teaching the limitations cited above for 
claims 22 and 32 and therefore, a prima facie case of obviousness is lacking because 
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Szeliski and Seago do not teach, disclose or suggest these limitations of the claims and 
are therefore deemed patentable. 

Examiner respectfully requests that applicant look to the response for claims 22 
and 32 above. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to ROBERTA PRENDERGAST whose telephone number 
is (571)272-7647. The examiner can normally be reached on M-F 8:30-5:00. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on (571) 272-7782. The fax phone number 
for the organization where this application or proceeding is assigned is 571 -273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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